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Introduction 

Wilburn Elementary School is a Title I school with a population of 
approximately 700 students. Wilburn operates on a year-round calendar 
that includes students on four different tracks or schedules, with each 
track having its own calendar. Under this configuration, the traditional 
180-day school year is divided into nine- week quarters with a three week 
break at the end of each quarter. Students are in school the same number 
of days as their counterparts in traditional schools; the days are simply 
redistributed throughout the year. Schedules are staggered so that, at any 
given time, three tracks are in school and one is on break. 

In 2010 Wilburn received a 5-year Teacher Incentive Fund (TIF) grant 
through the US Department of Education. This program supports efforts 
to develop and implement performance-based teacher and principal 
compensation systems in high-need schools. Goals include: 

1 . Improving student achievement by increasing teacher and principal 
effectiveness; 

2. Reforming teacher and principal compensation systems so that 
teachers and principals are rewarded for increases in student 
achievement; 

3. Increasing the number of effective teachers teaching poor, 
minority, and disadvantaged students in hard-to-staff subjects; and 

4. Creating sustainable performance-based compensation systems. 


Acknowledgement: We are grateful to Amy Germuth of 
EvalWorks, contractor on this project, for her extensive 
contributions to this evaluation and report. 


SUMMARY 

Implementation 

Based on interviews and 
observations, the four TAP 
principles were 
implemented with fidelity 
during the 2010-11 school 
year, with one exception — 
teacher evaluations. 
Feedback was slow, and 
teachers had concerns 
about the reliability of the 
ratings. The impact on 
teachers was negative. 

Student Outcomes 

Compared to 2009-10, 
reading and math 
proficiency improved in 
grades 3 and 5 in 2010-1 1 
but not grade 4. However, 
compared to proficiency 
before TAP began, only 
Grade 5 had improved 
reading proficiency and 
none of the grades (3, 4, 
and 5) had higher 
proficiency in math. 

Grade 4 had positive trends 
for Academic Change 
(growth) in both reading 
and math. 
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At Wilburn, TIF funding is being used to support the Teacher Advancement Program (TAP), 
freeing up Title I dollars that had been used previously. Wilburn is currently in its third year of 
implementing TAP. 

TAP is a comprehensive school reform initiative designed to attract, retain, and motivate quality 
teachers. The four principles upon which TAP is based are: 

• multiple career paths, 

• ongoing, applied professional growth, 

• instructionally-focused accountability, and 

• performance-based compensation (Agam, Reifsneider, & Warded, 2006). 

According to TAP descriptions, it is critical that all four principles are fully implemented to 
ensure success of a school or district. As noted above, a distinguishing feature of TAP compared 
to other school reform efforts is its merit pay, which is based on teacher performance and student 
academic growth. 

As a part of TAP, Wilburn teachers have the opportunity to participate in multiple career paths, 
receive rewards based upon their evaluations and student growth, receive job-imbedded 
professional development that is relevant, and receive evaluations based on an instructionally 
focused rubric. Anticipated long-term goals are an increase in teacher effectiveness, improved 
student learning of the curriculum (based on formative and summative assessment results), and 
closing achievement gaps. 

TAP was introduced in Wilburn Elementary School by staff vote in the 2008-09 school year to 
improve the effectiveness of Wilburn's teaching staff, offer additional support in implementing 
other initiatives such as the Sheltered Instruction Observation Protocol (SIOP®), and to decrease 
staff-development related absences, when possible. Wilburn decided not to take a full year for 
planning (including hiring, training, and certifying the leadership team), as is typical of most 
schools that adopt TAP, but combined their planning year with their first year of implementation. 

For the first year of TAP implementation mathematics was the focus improvement area. For 
Years 2 and 3 the focus has been on reading, given that students' proficiency in reading is very 
low. Improving students' reading abilities was also seen as one way to continue to help students' 
increase their mathematics proficiency levels. 

In November, 2010, in part due to low student performance, Wilburn was designated as one of 
four "Renaissance Schools" by the Wake County Public School System (WCPSS) as part of the 
district’s Race to the Top initiative. The Renaissance model includes signing bonuses, 
performance-based compensation, an infusion of technology, and an extra teacher per grade 
level. A selection process was held to staff the Renaissance schools. Current employees at each 
Renaissance school, including Wilburn, who wished to participate in the project, were required 
to reapply for positions for the 201 1-12 academic year. Those not wishing to reapply could 
request a transfer to another school. 


2 



TAP at Wilburn elementary- Year 3 


D&A Report No. 11.02 


Under the Renaissance Model, Race to the Top funds will be used to implement a signing bonus 
and performance-based compensation plan across the four schools. 'Variables used to determine 
teacher performance bonus pay include the overall academic success of the school, academic 
success of individual teachers and/or teachers' annual performance review. 1 2 

METHODOLOGY 

The data used to compile this report include TAP research, the program's logic model, and data 
gathered via meetings with the TAP Project Director at WCPSS, the Senior Director of Program 
Accountability in WCPSS Evaluation and Research department, interviews with the principal, 
and interviews with 24 teachers including specialists, arts and physical education teachers, and 
classroom teachers. Included in interviews with teachers were interviews with Mentor teachers 
as well as one Master teacher. Copies of the interview protocols used are included in the 
Appendix. Additional data were collected via document review including a review of the TAP 
manual, a description of the program components, the TAP lesson rubric, and the TAP teacher 
evaluation rubric. Test data for 2007-08, 2008-09, and 2009-10 by grade and area (Reading and 
Mathematics) are included in this report; 2010-11 data are not included as they are not available 
as of the date of this report. 

TAP IMPLEMENTATION AT WILBURN 

Tables 1-4 compare the four TAP components and describe implementation of these components 
at Wilburn. As is described, Wilburn implemented TAP with fidelity for the 2010-11 school 
year across all four principles. 

In terms of multiple career paths, TAP offered the opportunity for teachers to become mentor 
and master teachers without leaving the teaching profession. For the 2009-10 school year, eight 
Mentors were hired from within the school based on an interview process and the meeting of criteria set 
by TAP. 


1 Wilburn staff will receive only the sign-on bonus with Race to the Top funds. The performance-based 

compensation will come from the Teacher Incentive Fund Grant. 

2 Determinations for performance -based compensation for Wilburn staff will follow the eligibility criterion 

presented in the 2010 WCPSS Teacher Incentive Fund grant application. 
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Table 1 

Implementation - Multiple Career Paths 


TAP Program Description 

TAP Implementation at Wilburn 

Teachers advance 
professionally without having 
to leave the classroom. 

Based on interest, ability, and accomplishments, teachers are given the 
opportunity to advance professionally by becoming Mentor teachers. 
This new career path means teachers do not have to leave the 
classroom to continue to work with students and have more 
opportunity to work with their colleagues. As a Mentor teacher, 
teachers take on additional responsibilities, including leadership 
responsibilities, and receive supplemental pay. 

Master and Mentor teachers 
are chosen through a 
competitive, performance- 
based selection. 

For the 2009-2010 school year, eight Mentors were hired from within 
the school based on an interview process (with questions developed by 
TAP about designing, planning, and implementing instruction; 
classroom management; and working with teaching teams and 
parents). All Mentor teachers were required to be working towards a 
Master's degree and to have earned a 3.5 or higher average on a scale 
of 1 to 5 as part of the prior years' evaluations using the TAP 
Instructional Rubric. The Master teachers from 2009-10 continued in 
the 2010-11 school year, until one left in March to take an 
administrative position. After an extensive local search another Master 
teacher was hired in April. Flowever, she did not receive training until 
June, and thus could not perform many of Master teacher duties. 

Master and Mentor teachers 
have expert curricular 
knowledge and outstanding 
instructional skills and work 
effectively with others. 

Master and Mentor teachers design and deliver professional 
development to cluster groups and individual teachers while 
determining content knowledge and pedagogical skills that should be 
shared/enhanced as paid of cluster meetings. Both are paid of the 
leadership team and participate in setting school goals, supporting 
professional development, and using data to assess how students are 
progressing and what strategies are needed to improve achievement. 

Master and Mentor teachers 
set specific annual student 
learning goals. 

The professional development provided in cluster meetings focused on 
reading. Reading was selected as the focus area based on 2009-10 
reading scores. Strategies that were emphasized were based on specific 
analyses of End-of-Grade data, MAP data, and formative classroom 
assessments. Teachers presented student data at weekly cluster 
meetings and discussed findings. Teachers also collected data from 
their assessments of students on learning skills emphasized during 
cluster meetings and shared these data then as well. 

Master and Mentor teachers 
oversee TAP activities and 
provide leadership and support 
through group and individual 
coaching. Ongoing classroom- 
based support is provided 
through team teaching and 
classroom demonstrations. 

Master and Mentor teachers met weekly to determine which strategies 
and skills to emphasize in order to increase students’ reading 
achievement. Strategies were field-tested with Wilburn students at 
different grade levels before they were shared with other teachers. In 
addition, Master and Mentor teachers offered team teaching, classroom 
demonstrations, and feedback as paid of pre- and post-conferences, to 
help improve their colleagues' teaching although few teachers utilized 
team teaching or classroom demos. 

Master and Mentor teachers 
conduct teacher evaluations. 

Master and Mentor teachers, along with the principal and assistant 
principal, were responsible for conducting evaluations. Each teacher 
was evaluated six times (2 announced, 4 unannounced), using a 
structured evaluation rubric developed by TAP called the TAP 
Instructional Rubric. 
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As was described above, multiple career paths were implemented at Wilburn for the 2010-11 
school year. These included opportunities for two teachers to serve as Master teachers, and 
multiple teachers to serve as Mentor teachers. 

Table 2 


Implementation - Ongoing, Applied Professional Growth 


TAP Program Description 

TAP Implementation at Wilburn 2010-11 

Teachers advance 
professionally without having 
to leave the classroom 

Teachers spent 90 minutes per week in professional development as 
part of track-level cluster team meetings. While teachers were away 
from their classroom, teacher assistants provided instructional practice 
to students. 

Professional development is 
provided on a regular basis (1 - 
2 hours weekly). 

Professional development occurred during the school day for 90 
minutes each week. Teachers also met regularly outside of cluster 
meetings (such as at grade level meetings) to discuss the professional 
development they had received. 

Professional development is 
job-embedded, collaborative, 
and is focused on instructional 
practices based on specific 
students' instructional needs. 

TAP professional development was based on school, class, and teacher 
data and feedback and employed research-based strategies that had 
been field-tested and shown to be successful with Wilburn students. 
Professional development was sustained; follow-up and evaluation 
were also provided. 

Principal, Mentor, and Master 
teachers receive training and 
certification from TAP for 
conducting professional 
growth activities and teacher 
evaluations effectively. 

The principal and Master teachers received four days of training in 
August 2008 on "Preparing to Become a TAP School", which focused 
on the four principles of TAP and explained the TAP Instructional 
Rubric. They also received additional training in "Preparing to Become 
a Certified TAP Evaluator" and Becoming a Certified TAP Evaluator." 
New Mentor teachers received this training last summer to prepare 
them to conduct evaluations during the 2010-11 school year. 

Professional development 
allows teachers to learn new 
strategies that improve quality 
of instruction. 

The 2010-11 emphasis of professional development was on reading. 
Master and Mentor teachers met weekly to review data to determine 
the best strategies to meet students' needs and improve their reading. 
Strategies were field-tested with Wilburn students to ensure that they 
worked and then shared with teachers at track-level cluster meetings. 

Evaluations are based on TAP 
standards/rubrics and student 
growth. 

TAP has developed a detailed rubric for evaluating teachers' lessons. 
This rubric includes 19 indicators ranging from lesson planning to 
problem-solving. All teachers are trained in the content of this rubric. 
All evaluation components are linked to this instructional rubric. 

Each teacher is evaluated 4-6 
times a year with announced 
and unannounced evaluations. 
Pre-conferences are conducted 
prior to upcoming evaluation. 

Each teacher was evaluated 6 times in 2010-1 1 with 2 announced 
observations and 4 not announced. Pre-conferences were provided 
prior to the two announced evaluations. 

Feedback on observations is 
provided: post-conferences 
followed the evaluations to 
discuss needed refinements. 

All evaluations ended with a post-conference where teachers were 
provided detailed information about their lesson and how it was rated. 
Unfortunately, many post-conferences were not conducted in a timely 
manner. 

Group and individual coaching 
and demonstration lessons are 
regularly conducted by Mentor 
and Master teachers. 

Group coaching was conducted through weekly cluster groups with 
individual coaching available to teachers on an as-needed basis. As 
part of clusters, demonstration lessons were provided. 
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Although multiple professional development opportunities were provided to teachers during the 
2010-11 school year, teachers reported that evaluation feedback was not provided in a timely 
manner, thus reducing how instructive it could have been. 

Table 3 


Implementation - Instructionally-Focused, Performance Based Accountability 


TAP Program Description 

TAP Implementation at Wilburn 

Teachers are held accountable 
for meeting the TAP Teaching 
Skills, Knowledge, and 
Responsibilities Performance 
Standards and arc evaluated 
based on those standards. 

The TAP Evaluation Rubric contains 19 indicators that are detailed, 
specific, and focused on research-based best practices for lesson 
planning and delivery. A weighted point system is used to measure 
each indicator as "exemplary", "proficient", and "needs improvement". 

Teachers are evaluated based 
on academic growth of their 
students. All teachers arc 
evaluated collectively based 
on the learning growth of all 
students in the school. 

In grades K - 3 and for art, music, physical education, and special 
education teachers, where End-of-Grade test scores are not available, 
the quality of classroom instruction and school-wide student growth 
each represent 50% of a teachers' evaluation weight. In grades 4-5, due 
to the availability of End-of-Grade test scores, quality of classroom 
instruction is weighted 50%, the teachers’ student growth is weighted 
30%, and entire school growth is weighted 20%. 

Announced and unannounced 
observations are conducted by 
trained and certified 
evaluators. Pre-conference 
sessions are offered with 
announced observations and 
post-conferences are offered 
after all observations to 
discuss reinforcements and 
refinements. 

Six observations (two announced and four unannounced) were 
conducted in each classroom. Pre-conferences were conducted prior to 
announced observations. Post-conferences followed all observations. 


Instructionally-focused performance-based accountability was implemented in 2010-11 as part of 
TAP. This included holding teachers responsible for TAP Teaching Skills, Knowledge, and 
Responsibilities Performance Standards and evaluating teachers on the academic growth of their students. 
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Table 4 


Implementation - Performance-Based Compensation 


TAP Program Description 

TAP Implementation at Wilburn 

Compensation is provided to 
all teachers based on multiple 
evaluations of teaching and 
student growth at the 
classroom level and school 
levels. 

Performance-based compensation was implemented in the 2010-11 
school year based on 2009-10 data. 

The performance awards are 
recommended to be allocated 
in the following manner: based 
on performance standards 
(50%), individual classroom 
achievement (30%), and 
school-wide achievement 
growth (20%). 

Only grades 4-5 have EOG scores that will be used to measure student 
growth. They will provide 30% of the student performance measure 
for all teachers in the school and an additional 20% for K-3 teachers, 
and art, music, physical education, and special education teachers. In 
grades K-3 and for art, music, physical education, and special 
education teachers, where End-of-Grade test scores are not available, 
the quality of classroom instruction and school-wide student growth 
each represent 50% of a teachers' evaluation weight. 

Salary augmentation is given 
to Master and Mentor teachers 
according to their additional 
responsibilities and authority. 

Salary augmentation was given to Master and Mentor teachers in 
2010-11 according to their additional responsibilities and authority. 
Master teachers are paid a salary differential of $9,900 per year; 
Mentor teachers are paid a salary differential of $5,000 per year. 


Performance-based compensation was implemented as part of TAP in 2010-11 with teachers 
receiving compensation based on their students’ academic growth. In addition. Master and 
Mentor teachers received supplements for their work as part of TAP. 


TEACHER PERSPECTIVES ON TAP IMPLEMENTATION 

Based on multiple teacher interviews conducted in May 2011, teachers had both positive and 
negative things to say about TAP. For example, all teachers reported engagement in critical 
professional learning activities and other practices to guide their planning and lessons, such as: 

• Engaging in regular student-focused professional development; 

• Using more formative assessments to assess student understanding and identify areas 
where re-teaching may be needed; 

• Analyzing and reviewing data to guide instructional planning and practices; and 

• Learning new strategies that improved their quality of instruction. 

However, it is clear that there are some areas in which TAP implementation could be improved. 
Many teachers had concerns about some areas. These areas include: 

• Providing more timely communication regarding evaluation findings; 

• Implementing a formal feedback mechanism where teachers can share their concerns 
with the Leadership Team about TAP implementation; 
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• Reducing or attempting to reduce teachers' time away from their class as much as 
possible; 

• Ensuring that Cluster Meetings are meaningful to all teachers for the full 90 minutes; and 

• Working with teachers to determine the best possible schedule for Cluster Meetings. 

Additional concerns that were voiced by teachers during May 2011 interviews were less about 
TAP but still important to note. These include concerns about student behavior and the impact of 
being a Renaissance School: 

• Teachers in the upper grades, in particular, were worried that student behavior was 
increasingly left unchecked by administrators and that TAP has taken over such that 
administrators have little time available to address behavior problems among students. In 
the past Wilburn had used the Positive Behavioral Interventions and Supports. (PBIS) model 
for directing student behavior, but many felt that TAP had "over-ridden" the use of this 
model. 

• Being named a Renaissance School negatively affected morale, especially for teachers 
who had not yet been interviewed for jobs or knew where they will be placed the next 
year. Many teachers noted that it was hard to understand why students' test scores had 
dropped for the 2009-10 school year even with TAP in place, but many said they came 
back with an even stronger commitment to TAP. However, the announcement in 
November 2011 that Wilburn would be a Renaissance School and that all personnel had 
to reapply for their jobs was an incredibly negative blow to the school. As one teacher put 
it, being named a Renaissance school "has been a total disaster." One teacher summed it 
up by sharing, "TAP is stressful enough and Renaissance has taken over everything." 
Some teachers were planning to leave because they did not want to be in limbo about 
their jobs for the next year and others because they were leery of a new administration. 
However, some decided to reapply to Wilburn as they hoped that TAP would be 
continued at the school and that it would be remodeled to address their major concerns 
(time spent out of the classroom, lack of formal feedback mechanisms, and lack of post- 
conferences and true reflection/discussion around observations). 


TAP IMPACT ON STUDENT ACHIEVEMENT 

As noted previously, TAP is designed to increase the overall achievement of students by 
increasing teachers' effectiveness. However, Table 5 and Figure 1 show this has only occurred in 
reading at grade 5 over time. Grades 3 and 4 had similar or lower proficiency rates in 2010-11 
than before they began TAP (2007-08). Across grade cohorts, patterns were more positive. 

• In Year 2, the percentage of students at or above grade level in reading stagnated (Grades 
4 and 5) and decreased at Grade 3. 

• In Year 3, the percentage of students at or above grade level in Grade 5 increased slightly 
(3.6 percentage points) and in Grade 3 increased dramatically (13.4 percentage points). 
Unfortunately, in Grade 4 this percentage decreased dramatically (13.2 percentage 
points). (Based on results within grades in Table 5.) 
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• When examined across grades, we see that those in grade 3 in 2007-08 increased in 
reading proficiency rates in 2008-09 in grade 4 and stayed fairly stable in 2009-10 in 
grade 5 (see red bold font). The 2008-09 cohort of third graders actually increased in 
proficiency in reading in both grades 4 and 5 (see green italics font). The 2009-10 cohort 
of third grades was quite low compared to the rest, and improved by 6.7 percentage 
points in proficiency in 2010-11 (see underlined numbers). 


Table 5 

Reading: Percent of Students At or Above Grade Level 


Grade 

2007-08 

2008-09 

2009-10 

2010-11 

(TAP Year 1) 

(TAP Year 2) 

(TAP Year 3) 

3 

54.1 

58.2 

40.1 

53.5 

4 

49.5 

62.3 


46.8 

5 

54.6 

61.2 

61.9 



Figure 1 

Reading: Percent of Students At or Above Grade Level 



■ 3 

4 

■ 5 
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In mathematics, student proficiency was not higher at any grade in 2010-11 than before TAP 
began (although Grade 5 is similar). Students made gains in the first year of TAP when 
mathematics was the focus for that year. However, in 2009-10 (Year 2 of TAP) the percentage of 
students scoring at or above grade level for Grades 3, 4, and 5 declined considerably (Table 6 
and Figure 2). In Year 3, the percentage of students scoring at or above grade level increased in 
Grades 3 (9.8 percentage points) and Grade 5 (5.4 percentage points), but dropped in Grade 4 
(6.8 percentage points). The positive patterns of improvement for the 2007-08 and 2008-09 third 
grade cohorts in reading were not evident for mathematics (see bold and italics). On the other 
hand, the percentage of 2009-10 third graders who reached proficiency in 2010-11 as fourth 
graders did increase (see underlining). 


Table 6 

Mathematics: Percent of Students At or Above Grade Level 


Grade 

2007-08 

2008-09 

2009-10 

2010-11 

(TAP Year 1) 

(TAP Year 2) 

(TAP Year 3) 

3 

75.7 

76.4 

51.3 

71.1 

4 

77.9 

79.2 


62.4 

5 

68.5 

75.5 

62.7 



Figure 2 

Mathematics: Percent of Students At or Above Grade Level 


90 



2007-08 2008-09 2009-10 2010-11 
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The NC ABCs Academic Change scores are based on changes 
in individual student’s scores from the previous year. (See 
box for more details on the definition.) These “growth 
scores” provide another way to look at schools’ achievement, 
and they reveal a somewhat different pattern for Wilburn. 
Table 7 and Figure 3 show that increases in the percentage of 
students meeting their growth targets were evident at both 
grades 4 and 5 in 2010-1 1 compared to the prior two years. 
This is particularly encouraging for grade 4 students, who 
actually reached “high growth” if we apply the school 
standard of 60% or more to the grade levels. Grade 5 is still 
below that standard for high growth, but also considerably 
better in 2010-1 1 than previously. 

Academic Change scores indicate that 4th grade teachers, 
while not able to move as great a percentage of students to 
performing at or above grade level, actually helped more 
students reach their growth targets than was true at grade 5. 
Patterns of improvement over time are similar at the two 
grades. Please note that growth data are not available for 3rd 
graders after 2008-2009 (when the fall pretest was dropped 
from the state accountability system). 


State ABCs Growth Model 
and Academic Change Scores 

Students are expected to do at 
least as well this year as they 
have in the past, compared to 
other NC students who took the 
same statewide test in the year 
standards were set (usually the 
first year the test was given). 
Academic Change scores are 
calculated for each student who 
has scores from the current and 
past school year. 

Schools meet “expected 
growth” if students, on the 
average, show a year of growth 
in a year’s time. Schools meet 
“high growth” if 60 % of 
students in a school meet their 
individual growth targets. 
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Table 7 

Reading: Percent of Students Meeting Growth Targets 
(Academic Change >= 0) 


Grade 

2008-09 
(TAP Year 1) 

2009-10 
(TAP Year 2) 

2010-11 

(TAP Year 3) 

4 

59.1 

56.9 

69.3 

5 

38.1 

34.7 

49.5 


Note: Grade 3 Academic Change cannot be calculated because the state no longer offers a Grade 
3 Pretest. 


Figure 3 

Reading: Percent of Students with Academic Change >= 0 



2008-09 2009-10 2010-11 
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In mathematics, increases in the percentage of students meeting their growth targets were evident 
at both grades 4 and 5 in 2010-11 compared to 2009-10, but not when compared to 2008-09. 

• Grade 4 students showed considerable improvement in 2010-1 1 compared to 2009-10 
(with a 21 percentage point improvement in students making growth), but the 
improvement compared to 2008-09 was a more modest; six percentage points. Grade 4 
reached “high growth” if we apply the state standard used for schools. 

• Grade 5 patterns were less positive, with a lower percentage of students reaching their 
growth targets in 2010-11 than in 2009-10. Percentages have actually changed very little 
across the three years. Grade 5 did not reach the “high growth” standard of 60%. 
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Table 8 

Mathematics: Percent of Students Meeting Growth Targets 
(Academic Change >= 0) 


Grade 

2008-09 
(TAP Year 1) 

2009-10 
(TAP Year 2) 

2010-11 

(TAP Year 3) 

4 

55.0 

40.2 

61.4 

5 

47.6 

50.5 

46.5 


Figure 4 

Mathematics: Percent of Students with Academic Change >= 0 


70 



0 

2008-09 2009-10 2010-11 


Math patterns suggest grade 4 teachers did a better job of promoting growth for more of their 
students in 2010-11 than in 2009-10 despite reading being the focus area. The same was not true 
for grade 5. 


SUMMARY AND DISCUSSION 


IMPUEMENTATION 

Based on interviews and observations, it seems reasonable to conclude that the four principles 
upon which TAP is based were implemented with fidelity during the 2010-11 school year, with 
one exception — teacher evaluations. Slow feedback was likely less helpful than it could have 
been. While there may be legitimate reasons for why this happened so frequently, the effect was 
quite negative on teachers, for whom the evaluations most matter and are seen as high stakes. 
The doubts of some teachers about the reliability of the ratings also would likely decrease the 
extent to which they took the feedback to heart. 

Although teacher interviews, conducted in May 2011, suggest that some teachers at Wilburn 
were critical of TAP, especially regarding usefulness and reliability of evaluations, it is 
important to note that these interviews were conducted during the period in which teachers were 
re-applying to teach at Wilburn or interviewing for new teaching assignments. Much was up in 
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the air, including which teachers would be returning to Wilburn, and where other teachers would 
be headed, or if they would even have jobs for the next year if they were new or non-tenured 
teachers. 

There are many positive aspects of TAP that teachers did note, especially around their own 
professional development. Teachers reported: 

• Engaging in regular student-focused professional development; 

• Using more formative assessments to assess student understanding and identify areas 
where re-teaching may be needed; 

• Analyzing and reviewing data to guide instructional planning and practices; and 

• Learning new strategies that improved their quality of instruction. 

However, teacher interviews do point to the need to better support teachers' involvement with 
TAP, including providing them formal mechanisms where they may discuss their perceptions of 
TAP. As noted, in 2010-11 teachers were very troubled that the TAP Leadership Team were not 
hearing the concerns they had about such things as time away from students, how post- 
conferences were not being conducted in a timely manner, and the reliability or lack thereof of 
observation ratings. Involving more teachers in something like the school’s Leadership Team 
where they can have input into TAP implementation may go a long way to maintaining and/or 
increasing buy-in of TAP. 

STUDENT OUTCOMES 

Of critical importance is how students performed in 2010-11, the third year of TAP. A recent 
independent study by Glazerman, McKie, and Carey (2009) found that positive student impacts 
with TAP may require considerable time to take hold. Other research studies (Schacter et al., 
2002; Schacter et al., 2004; Solomon et al., 2007; and Springer et al., 2008) have shown that 
favorable results in terms of students' performance and growth can occur in 1-3 years. These 
studies also documented considerable variability in outcomes across state, site, and grade level, 
as well as level of implementation. 

bindings from past and current achievement data indicate that in Year 3 of TAP, Wilburn 
teachers in Grades 3 and 5 were able to reverse troubling trends in reading proficiency rates.. 
While grade level trends are important, they do not take into account changes in student cohorts 
which can occur. Within Grade 4, proficiency declined but growth trends were positive. Across 
grades, the percentage of students coming into grade 4 scoring at grade level was considerable 
lower than for those entering grade 5, which helps to explain these results. 

Potential reasons for the improvements in reading proficiency at Grades 3 and 5 but substantial 
decreases (Grade 4) in the percentage of students scoring at or above grade level are hard to 
pinpoint. One factor is that the 2010-11 fourth-grade students came in with lower proficiency 
overall than prior groups, as well as lower proficiency than was evident for grades 5. In addition, 


15 



TAP at Wilburn elementary- Year 3 


D&A Report No. 11.02 


fourth grade teachers could have been better at promoting growth for students already at grade 
level than for those starting out below grade level. It is also possible that delays in feedback and 
concerns about reliability of ratings were greater at grade 4 than the other grades. 

Mathematics patterns were less positive than reading patterns, which may relate to the fact that 
math is no longer the main focus area for TAP. 

Another year of data will make patterns easier to discern. The impact of the change in staffing 
brought on by Renaissance status will be interesting to watch. Losing teachers with TAP 
experience could have a negative impact if teachers were just about to break through to make big 
differences in their practice. The main question with respect to student achievement data is 
whether reading scores will increase in 201 1-12, and what effect such a gain, if it occurs, may 
have on math scores. 


RECOMMENDATIONS 

Recommendations for future years of TAP include the following: 

• Greater teacher involvement with TAP, including providing them formal mechanisms 
where they may discuss their perceptions of TAP. 

• More timely follow-up to observations (by meeting with teachers within one week of 
conducting observations) in order to debrief with teachers about the rating he or she 
received, the reasoning behind it, and suggestions for improvement. 

• Reduce the time, to as large a degree as possible, that teachers are out of the classroom on 
any given day, including as part of TAP training. 

• Consider teachers’ patterns of growth and performance change at grades 3 through 5 in 
making teacher retention decisions. At grades K-2, review data available from reading 
book levels, math strands, and other available assessment measures. For all grades, 
consider the extent to which teachers were implementing TAP with fidelity. 
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APPENDIX: INTERVIEW PROTOCOLS 

Wilburn Elementary School TIF Evaluation 
Semi-Structured Interview Protocol - Teachers 


1. Do you consider yourself a TAP school - why or why not? 

2. How helpful are cluster meetings? What have they done for you (e.g., improved instruction, 
increased focus on reading achievement, improved collaboration)? What about team 
meetings? 


3. The four elements of TAP are: 

• Multiple career paths 

• Ongoing applied professional growth 

• Instructionally focused accountability 

• Performance-based Compensation 

Let’s talk about each one - do you think these elements are evident in your school and 
how do you feel about them? 


4. What would you like TAP to look like at your school to best promote student achievement? 


5. How would you describe teachers’ buy-in of TAP this year? What about compared to last 
year? 


6. How do you think you will perform as a school this year? 


7. What more could be done to improve student achievement at your school? 


8. How many of you want to stay here next year? 
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Wilburn Elementary School TIF Evaluation 
Semi-Structured Interview Protocol - Mentor Teachers 


1. What made you want to be a Mentor Teacher? 

2. Please describe your role as a Mentor Teacher. 

3. The four elements of TAP are: 

• Multiple career paths 

• Ongoing applied professional growth 

• Instructionally focused accountability 

• Performance-based Compensation 

Let’s talk about each one - do you think these elements are evident in your school and 
how do you feel about them? 


4. What would you like TAP to look like at your school to best promote student achievement? 


5. How would you describe teachers’ buy-in of TAP this year? What about compared to last 
year? 


6. How do you think you will perform as a school this year? 


7. What more could be done to improve student achievement at your school? 


8. How many of you want to stay here next year? Want to continue as TAP Mentor Teachers? 


19 



TAP at Wilburn elementary- Year 3 


D&A Report No. 11.02 


Wilburn Elementary School TIF Evaluation 
Semi-Structured Interview Protocol - Master Teacher 


1. What made you want to be a Master Teacher? 

2. Please describe your role as a Master Teacher. 

3. The four elements of TAP are: 

• Multiple career paths 

• Ongoing applied professional growth 

• Instructionally focused accountability 

• Performance-based Compensation 

Let’s talk about each one - do you think these elements are evident in your school and 
how do you feel about them? 


4. What would you like TAP to look like at your school to best promote student achievement? 


5. How would you describe teachers’ buy-in of TAP this year? What about compared to last 
year? 


6. How do you think you will perform as a school this year? 


7. What more could be done to improve student achievement at your school? 


8. How many of you want to stay here next year? Want to continue as TAP Mentor Teachers? 
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Wilburn Elementary School TIF Evaluation 
Semi-Structured Interview Protocol - Principal 


1. The four elements of TAP are: 

• Multiple career paths 

• Ongoing applied professional growth 

• Instructionally focused accountability 

• Performance-based Compensation 

Let’s talk about each one - do you think these elements are evident in your school and how 
do you feel about how they are working? 

2. How helpful are cluster meetings? What have they done for you (e.g., improved instruction, 
increased focus on reading achievement, improved collaboration)? What about team 
meetings? 

3. Do teachers have PLT time beyond their cluster and grade level meetings? If one of those 
functions as their PLT meeting, is the format and content any different than the WCPSS 
model (which is largely based on DuFours)? 


4. How would you describe teachers’ buy-in of TAP this year? What about compared to last 
year? 

5. What structures are in place to monitor student achievement? (Ask about all assessments, 
timing, and use) 

6. How do you think you will perform as a school this year? 

7. What more could be done to improve student achievement at Wilburn? 
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